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Abstract 

This research paper presents a database management system for tracking course assessment data 
and reporting related outcomes for program assessment. It improves on a database system 
previously presented by the authors and in use for two years. The database system presented is 
specific to assessment for ABET (Accreditation Board for Engineering and Technology) accreditation 
effort of a CIS (Computer Information Systems) department, but could be easily adapted to any 
program assessment. The relevance of tracking course assessment data and its role in reporting the 
outcomes for assessment is discussed. Issues of consistency, efficiency, flexibility, and reusability are 
discussed and sample data and reports are presented. This paper concludes with a discussion of the 
merits of the database management system as a tool in program assessment. 
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1. INTRODUCTION 

Program assessment has become one of the 
highest priorities in higher education and a key 
component of accreditation efforts for any 
academic program. Program review and 
accreditation efforts for organizations such as 
ABET and AACSB require the tracking of data 
from 

such sources as questionnaires from alumni and 
feedback from advisory boards. Another key 
ingredient for assessing program outcomes is 
the performance of students in the required 
coursework in the program. Outcomes of any 


instrument used to evaluate student 
performance, such as homework, projects, 
presentations, course exams, and national 
exams, can be used for course or program 
assessment. Course objectives can be 

evaluated using objective measures and this 
data can be aggregated to assess the program 
outcomes. To support the gathering and 
analysis of this data, a relational database was 
designed and implemented in Microsoft Access. 
This paper describes the mapping of course 
learning objectives for a CIS department's 
courses to both the program outcomes and ABET 
outcomes, the development of a relational 
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database to support the assessment of courses, 
and an ongoing evolution of that database 
system. 

2. LITERATURE REVIEW 

A review of the literature shows that while there 
is a wealth of research on the general topic of 
program assessment, there has been a relatively 
limited number of efforts to develop tools and 
software to help streamline the time-consuming 
and burdensome process of data collection and 
analysis for program assessment. 

Spurlin reports on the development of a 

database for collection and analysis of data of an 
engineering program for ABET accreditation 
(Spurlin, Rajala, Lavelle, & Hoskins, 2002). 
However, it is not clear if the system allows for 
easy input of data from exams and projects and 
easy generation of assessment reports. Harding 
(2005) reports on the development of a 

prototype web-based assessment system for an 
ABET accredited engineering program. The 
author indicates that the student developed 
prototype had a number of problems but 
provides a good base to build upon. 

Poger describes a web-based system developed 
for online evaluation of courses for a Computer 
Science department (Poger, Kamari, Chuah, & 
Ricardo, 2005). The system is used to compile 
data from student prerequisite skills surveys, 
course objective student surveys, faculty course 
objectives assessment, and student exit 
surveys. 

Booth proposes a database template that, 
among other things, provides a mechanism to 
map program outcomes and course objectives to 
ABET outcomes (Booth, Preston, & Qu, 2007a). 
Booth also proposes requiring students to 
submit assignments via the Web and a method 
for extracting assessment metadata from these 
assignments. Booth presents a prototype of the 
implementation of the template in a second 
paper (Booth, Preston, & Qu, 2007b). 

Urban-Lurain proposes a database system that 
includes a newly defined extensible Educational 
Metadata Language- EdML(Urban-Lurain, Ebert- 
May, Momsen, McFall, Jones, Leinfelder, & 
Sticklen, 2009). EdML enables assessments to 
be tagged based on taxonomies, psychometrics, 
and other data to facilitate analysis. 

Segall presents a database driven system for IS 
Curriculum Assessment using the national ISA 
exam (Segall, Ghosh, & Morrell, 2009). The 
system provides for two-step mapping from ISA 
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exam questions to course objectives via the 
2002 IS Model curriculum Learning Units (LU). 

Essa (2010) proposes the ABET Course 
Assessment Tool (ACAT) as his Masters thesis. 
ACAT is a web-based tool that allows Faculty to 
input assessment data and generate a report in 
pdf format. This work appears to take a similar 
approach to the database design taken by 
Morrell (Morrell, Morris, & Haga, 2009). 

3. BACKGROUND 

The CIS Department faculty at Metropolitan 
State College of Denver debated for months 
before finally agreeing on a set of program 
outcomes prior to the initial pursuit of ABET 
accreditation. The final result was a set of 10 
program outcomes, which are listed in Table 1. 
Subsequently, many additional months were 
spent mapping: 1) CIS course learning 
objectives to these CIS Department outcomes; 
and 2) CIS course learning objectives to the 10 
ABET outcomes. The ABET outcomes are listed 
in Table 2. 

This work was performed by a CIS Department 
taskforce. During these meetings, a faculty 
member came up with the idea of developing a 
database management system to record and 
analyze assessment data for courses. A 
collaborative process lead to the design of this 
relational database. 

The goals for this database design were as 
follows: 

a) The design must be sufficiently flexible 
to allow tracking of any instrument used 
in evaluating students while allowing the 
faculty to determine the scale and level 
of the items measured. 

b) The design must provide for a control of 
consistency of the data captured. 

c) The design must provide a mechanism 
for mapping the course evaluations to 
program and ABET outcomes. 

d) The design must be portable and 
reusable for subsequent accrediting 
efforts. 

e) The data entry for the system must be 
sufficiently utilitarian to be acceptable by 
faculty and staff. 
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4. DBMS DEVELOPMENT 

Initial Database Design and 
Implementation 

The initial database design was previously 
presented by Morrell at the Northeast American 
Society of Engineering Education (Morrell, 
2009). 

The Entity Relationship Diagram (ERD) for the 
improved database design is shown in Figure 1. 
The database entities are explained in Table 3. 
Looking at the design we can see the tables that 
relate to the two mapping processes described 
above. The mapping of CIS course objectives to 
the CIS Program Outcomes is handled by the 
CourseObjective, Po_Co, and 

Program_Outcomes tables. The mapping of CIS 
course objectives to the ABET Outcomes is 
addressed by the Course_Objective, Ao_Co, and 
Abet_Outcomes tables. The basic Course, 
Faculty, and Class data was imported from the 
college scheduling system. 

The assessment instruments, such as exams, 
quizzes, assignments, and projects were entered 
into the Eval_Instrument table. To facilitate 
access for faculty and staff, the database is 
stored on a network shared folder. Faculty have 
access to this folder and the Information 
Technology department configured access for an 
administrative assistant. Faculty can enter the 
data themselves or record the data on a paper 
form. The data from the latter can then be 
entered by an administrative assistant. 

After the assessment instruments are entered, 
faculty can enter evaluation items for each 
instrument. In terms of tracking students' 
success, a faculty member can choose one 
multiple choice question, a group of multiple 
choice questions, a short answer question, a 
whole project or part of one. In other words, 
the faculty member has total control over the 
granularity they use. 

Before any data is captured the Department 
must set up guidelines for entering the results of 
any item on any instrument. An example would 
be the countright attribute in the database - see 
Figure 1. If the instrument is an exam and the 
item is a subjective question, then the 
department must decide what "countright" 
means. The department agreed that a passing 
grade means that the student scored a seventy 
percent on that item. For multiple choice 
questions, the determination of passing is clear, 
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but for subjective items, the seventy percent 
measure is used. 

The class entity includes time attributes, which 
will allow for periodic review, analysis, and 
comparisons over any number of 
semesters/years. The database could also be 
easily modified to track curriculum changes and 
their effect on student outcome scores. 

Sample Database Form and Report 

A data entry form (see Figure 2) is used to 
facilitate data entry of the detailed assessment 
data. It also promotes consistency and 
accuracy. 

This form allows users to enter assessment data 
for each evaluation item (e.g. question 11) for 
each evaluation instrument (e.g. an exam). 
Therefore this form corresponds to the 
Eval_Instrument and Eval_Outcome tables (see 
Figure 1). The user first selects a class ID (CRN) 
from a drop-down box and information about 
this class is displayed so the user knows they 
are entering data for the correct class. They 
then enter the data for exams/projects as well 
as how many students were successful on the 
chosen questions. 

The data entry form emphasizes the granularity 
possible with this design. Looking at the bottom 
part of the form a faculty member can enter an 
Eval_ID of El for Exam 1 and then enter every 
question on that exam as an evaluation item. 
Entering each question as an evaluation item 
would only make sense if these questions can be 
mapped to the course and ABET objectives. 
Alternatively they could group five questions 
that pertain to one course objective and enter 
this as one row in the form. 

The CIS Assessment Data by ABET Outcomes 
report (see Figure 3) displays the detailed 
assessment data grouped by ABET outcomes 
a-h. Each outcome has a list of all courses, 
evaluation instruments, as well as the success 
rate for each evaluation item that pertains to 
that outcome. The Percent column indicates the 
percentage of students in a class who got a 
question correct. 

Second Database Design and 
Implementation 

The initial database system was used for one 
year and the information generated was used in 
the creation of the Self-Study report for ABET 
accreditation. After having had a year's 
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experience with the database management 
system, faculty expressed a few concerns: 

a) Regardless of whether an assessment 
item was a multiple choice question, an 
essay question, a minor homework 
assignment, or even a major project 
there was no way to "weight" the item - 
so for example a multiple choice 
questions was being counted as equal to 
a major project. 

b) There was no distinction between a 
lower level objective such as "Identify 
the components of a LAN" and a higher 
level objective such as design a 
LAN/WAN network. 

c) Faculty had trouble remembering what 
the assessment items were when 
reviewing reports later. 

This lead to the following modifications to the 
database: 

Weight is added to the Eval_Outcome table. 
This allows a professor to distinguish between 
the various types of exam questions (true/false, 
multiple choice, fill-in the blank, short answer, 
essay, program, coding, design problems, etc) 
and the various types of homework assignments 
(research papers, programming assignments, 
minor projects, major projects, etc). For 
example in the case of a database course, it is 
clear that a multiple choice question on an exam 
should not be weighted the same as a problem 
that requires students to create an ERD from a 
set of several business rules. The ERD should 
have a much higher weight than the multiple 
choice question when it comes to determining if 
the course objective, and therefore a program 
outcome, is met. An example of weighting for 
such a class, a multiple choice question could be 
assigned a weight of 1, a short answer question 
a weight of 3, and ERD a weight of 5 and a 
major project a weight of 10. 

BloomLevel is added to both the 
Course_Objective and EvaLOutcome tables. 
When developing/modifying course learning 

objectives, faculty frequently use the Bloom 

levels. In fact, the college-wide Curriculum 
Committee for this college checks these 
objectives using Bloom levels. Adding the 
attribute to the EvaLOutcome table allows 
faculty to match evaluation items with the 

learning objectives in the course syllabus. It 
could also be used by the course coordinator 
and/or curriculum committee when reviewing 
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assessment to identify possible problem areas 
(i.e. Faculty trying to measure higher level 
objectives with all multiple choice questions). 

ItemType is added to the EvaLOutcome table. 
It allows faculty to indicate the type of 
evaluations item, such as multiple choice 
question, short answer question, assignment, 
etc. This is helpful when reviewing the 
assessment data at the end of a semester. 

ItemDesc is added to the EvaLOutcome table. 
It allows faculty to indicate the concept/skill 
covered by the evaluations item. Using the 
database class example again, while ItemType 
could indicate a multiple choice question, 
ItemDesc for example could tell us that it was a 
question related to foreign keys. This is helpful 
when reviewing the assessment data at the end 
of a semester. Otherwise the faculty member 
has to go back to the evaluation instrument, say 
an exam, and find the question to which this 
item pertains. 

5. DISCUSSION 

The initial database design in this paper was 
adopted by the authors' CIS Department and 
has now been used for 4 semesters. 

Dr. Janos Fustos, Chair of the department's 
ABET committee, provides the following 
evaluation of the database system: 

Being charged to coordinate the CIS 
program's ABET activities, I appreciate your 
work that provides us with this important 
tool. As accreditation agencies have moved 
to an outcome oriented program assessment 
it is vital for us to have a data source that 
records all the assessment activities and 
data that we can use to evaluate how far our 
program meets the aimed goals and criteria 
proposed by accreditation requirements. The 
database system that you've created allows 
us to capture the necessary data, generate 
reports at different granularity levels, and 
supports all of our documentation 
requirements including course revisions, 
curriculum modifications, annual analysis, 
regional and professional accreditation, and 
periodical program reviews. 

Course coordinators are required to, and 
responsible for, entering assessment data for 
their courses in the database each semester. 
The database system and its results, along with 
specific ways the system help to improve 
courses, was presented to the ABET Visiting 
Team in the Fall of 2009. The team responded 
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with very positive feedback. In addition, other 
departments seeking accreditation at the college 
have expressed a desire to have access to the 
database system. The reaccreditation process 
was successful. In fact, the department received 
an Assessment Achievement Award for its 
diligence in identifying course-embedded 
sources of evidence, articulating targets for 
student scores on program outcomes data, and 
engagement of the faculty. The award was 
accompanied by a monetary prize. Therefore the 
authors believe this bottom-up approach of 
assessing student performance in CIS courses 
played an important role in successful 
reaccreditation. This information was used in the 
creation of the ABET Self-Study Report. 
However, the system is much more flexible and 
adaptable than just this particular application. 
Indeed these are key characteristics of the 
design. The system could be used not just for 
ABET accreditation but also any other academic 
accreditation effort such as AACSB. In addition 
it could be used by any non-academic 
organization that captures numeric data for 
assessment 

Another strength of the design is that the 
expert, say a course coordinator, is in control of 
the chosen granularity. They can choose 
whether they want to track individual multiple 
choice questions, a group of multiple choice 
questions, a complete project, or a part 
thereof. In addition to the choice of granularity, 
an organization can choose which attributes to 
use. For example, they could initially choose to 
ignore Weight and BioomLevei. If necessary 
they could start capturing these items in later 
time periods. This flexibility reduces the barriers 
for the adoption and use of an assessment 
tracking system. 

The organization also decides how to handle 
such issues as what percentage constitutes 
passing a subjective evaluation item such as a 
short answer question. This department uses 
greater than or equal to 70% as a passing score 
and then counts the number of students that got 
this score. Another organization could choose a 
different value. 

How can the information from the generated 
reports be used? An example from a LAN/WAN 
class illustrates the potential. The faculty 
member noted from the reports that students 
answered a question about routing and 
autonomous systems poorly. More emphasis 
was given to this material in the following 
semester with a resulting improvement in 
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student results. Thus one key application of the 
system is to identify weaknesses in the coverage 
of the subject matter and to be able to address 
these issues in subsequent semesters. 

Another example of the use of the information 
captured by the system is the identification of 
anomalies. For example, a course may have a 
certain Bloom level but there are no evaluation 
items at this Bloom's level. The department 
could then evaluate how to improve assessing 
the students in this course. This illustrates the 
potential of the system to be used for 
addressing continuous improvement. 

In conclusion, the database design allows the 
tracking of course results at any level of 
granularity. If a department is in doubt about 
the level of detail required, the database can 
store all the items of all instruments used to 
determine course or program results. This is 
important to programs that undergo accrediting 
for multiple agencies such as ABET and AACSB. 
The mantra for this system is "If the data is 
sufficiently captured and organized, any 
reporting requirement on program results can be 
met". Hence the Department must invest the 
time and effort in developing the requirements 
for collecting the data. 

6. SUMMARY AND CONCLUSION 

Program assessment is now one of the highest 
priorities in higher education and has a vital role 
in program accreditation. Course assessment, 
which is an important focus in program 
assessment, is centered on collecting and 
reporting data related to course outcomes. 
Although there are various approaches to the 
latter task, this paper presents a system which 
is DBMS centered to accomplish this task. The 
system described in this paper is one that 
addresses the issues of consistency, flexibility, 
reliability, and reusability. The discussions 
above make a case that the system is technically 
and operationally feasible and that it meets 
criteria developed by the department for 
program assessment. The discussion of the 
evolution of the systems reveals the critical 
element of involvement of program constituents 
and the flexibility and utility of the system. 
Although the system was developed for a 
specific program for accreditation, it has the 
potential to be used by any similar organization 
(program or department) undergoing 
assessment for any purpose such as 
accreditation by agencies including ABET or 
AACSB. While the system is designed to be 
used primarily for tracking class assessment 
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data, it can be modified or adopted for tracking 
any quantitative metric used in assessment of 
an organization by an agency. The overall goal 
of such a system is to simplify, standardize, and 
organize the capture of data to provide a rich 
repository to support current and future 
assessment efforts. The system described in 
this paper meets that goal. 
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Appendix 


Table 1 - CIS Program Outcomes 

The program enables students to achieve (by the time of graduation) the following outcomes within 
the framework of professionally accepted Information Systems practices: 


SOI: 

Knowledge of basic information systems theory and concepts and the skills to apply this 
knowledge to the functional areas of business 

S02: 

Knowledge of project management tools and techniques as they apply to Information 
Systems projects 

S03: 

Knowledge of programming processes including planning, writing, testing, executing and 
debugging 

S04 


Knowledge of database design, development and management 

S05 


Knowledge of telecommunications and networking systems 

S06 


Knowledge of web-based systems 

S07 


Knowledge of operating systems 

S08 


Knowledge of how to create and utilize team approaches to problem solving 

S09 


Advanced knowledge in an IS area 

SO10: 

Ability to support the delivery and management of information systems 
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Table 2 - ABET Outcomes 

The program enables students to achieve, by the time of graduation: 



An ability to apply knowledge of computing and mathematics appropriate to the 

discipline; 


An ability to analyze a problem, and identify and define the computing requirements 

appropriate to its solution; 


An ability to design, implement and evaluate a computer-based system, process, 

component, or program to meet desired needs; 


An ability to function effectively on teams to accomplish a common goal; 


An understanding of professional, ethical, legal, security, and social issues and 

responsibilities; 


An ability to communicate effectively with a range of audiences; 


An ability to analyze the local and global impact of computing on individuals, 

organizations and society; 


Recognition of the need for, and an ability to engage in, continuing professional 

development; 


An ability to use current techniques, skills, and tools necessary for computing 

practices. 


An understanding of processes that support the delivery and management of 

information systems within a specific application environment. 
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Figure 1 - Database Design 
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Entities 

Description 

ABET_OUTCOMES 

All ABET outcomes. 

AO_CO 

Maps course objectives to ABET outcomes. 

CLASS 

Specific information for a given section offering of a course 

COURSE 

General information about each Course 

COURSE_OBJECTIVE 

All course objectives for a course 

EVAL_OUTCOME 

Individual items from a specific evaluation instrument 

EVALUATION_INSTRUMENT 

General information about a specific assessment 

FACULTY 

General information about the faculty member. 

PO_CO 

Maps course objectives to program outcomes. 

PROG RAM_OUTCOM ES 

All program outcomes for the CIS degree. 


©2011 EDSIG (Education Special Interest Group of the AITP) 
www.aitp-edsig.org /www.isedj.org 


Page 30 















Information Systems Education Journal (ISEDJ) 


9(7) 

December 2011 


Figure 2 - Data Entry Form 
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Figure 3 - Reporting Assessment Data by ABET Outcome 










CIS Assessment Data by ABET Outcomes 



ABET Outcome Course Number 

Course Objective # Course Objective 

Evaluation Item 

Percent 

a 

An ability to apply knowledg 

e of computing and mathematics appropriate to the discipline; 



3030 

6 

Search for and use Web 
development resources 

Discuss2 

75.00% 


3030 

6 

Search for and use Web 
development resources 

Topic2 

80.00% 


3030 

6 

Search for and use Web 
development resources 

Topicl 

85.00% 


3030 

6 

Search for and use Web 
development resources 

Search 

90.00% 

b 

An ability to analyze a problem, and identify and define the computing requirements appropriate to 
its solution; 



3030 

2 

Develop Web site 
specifications for any kind of 
Web site from a set of 
requirements 

q39 

94.74% 


3030 

2 

Develop Web site 
specifications for any kind of 
Web site from a set of 
requirements 

qis 

95.00% 


3030 

2 

Develop Web site 
specifications for any kind of 
Web site from a set of 
requirements 

q50 

100.00% 


3030 

2 

Develop Web site 
specifications for any kind of 
Web site from a set of 
requirements 

q49 

94.74% 


3030 

2 

Develop Web site 
specifications for any kind of 
Web site from a set of 

q48 

84.21% 
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